Diagnostic performance of GENEDIA W and ActiveXpress+ COVID-19 antigens tests among symptomatic individuals in Peru and The United Kingdom

Objectives In order to generate independent performance data regarding accuracy of COVID-19 antigen-based rapid diagnostic tests (Ag-RDTs), prospective diagnostic evaluation studies across multiple sites are required to evaluate their performance in different clinical settings. This report describes the clinical evaluation the GENEDIA W COVID-19 Ag Device (Green Cross Medical Science Corp., Chungbuk, Korea) and the ActiveXpress+ COVID-19 Complete Testing Kit (Edinburgh Genetics Ltd, UK), in two testing sites Peru and the United Kingdom. Methods Nasopharyngeal swabs collected from 456 symptomatic patients at primary points of care in Lima, Peru and 610 symptomatic participants at a COVID-19 Drive-Through testing site in Liverpool, England were analyzed by Ag-RDT and compared to RT-PCR. Analytical evaluation of both Ag-RDTs was assessed using serial dilutions of direct culture supernatant of a clinical SARS-CoV-2 isolate from the B.1.1.7 lineage. Results For GENEDIA brand, the values of overall sensitivity and specificity were 60.4% [95% CI 52.4–67.9%], and 99.2% [95% CI 97.6–99.7%] respectively; and for Active Xpress+ the overall values of sensitivity and specificity were 66.2% [95% CI 54.0–76.5%], and 99.6% [95% CI 97.9–99.9%] respectively. The analytical limit of detection was determined at 5.0 x 102 pfu/ml what equals to approximately 1.0 x 104 gcn/ml for both Ag-RDTs. The UK cohort had lower median Ct values compared to that of Peru during both evaluations. When split by Ct, both Ag-RDTs had optimum sensitivities at Ct<20 (in Peru; 95% [95% CI 76.4–99.1%] and 100.0% [95% CI 74.1–100.0%] and in the UK; 59.2% [95% CI 44.2–73.0%] and 100.0% [95% CI 15.8–100.0%], for the GENDIA and the ActiveXpress+, respectively). Conclusions Whilst the overall clinical sensitivity of the Genedia did not meet WHO minimum performance requirements for rapid immunoassays in either cohort, the ActiveXpress+ did so for the small UK cohort. This study illustrates comparative performance of Ag-RDTs across two global settings and considers the different approaches in evaluation methods.


Results
For GENEDIA brand, the values of overall sensitivity and specificity were 60. 4%

Introduction
The impact of the Severe Acute Respiratory Syndrome Coronavirus-2 (SARS-CoV-2), pandemic on healthcare has no precedents, stretching healthcare systems all around the world [1,2]. The development of rapid, and easy-to-perform diagnostic tools has been a priority, and various rapid diagnostic tests for the detection of SARS CoV-2 antigens (Ag-RDTs) have been developed since the pandemic and are currently available on the market. FIND lists 206 SARS CoV-2 Ag-RDTs that are currently marketed or in development (data accessed November 2021) [3]. Despite the myriad of Ag-RDTs that are currently commercially available, at the time of this study there are only a few studies that evaluate their clinical accuracy directly from patients' samples.
In Peru, at the beginning of the pandemic, only a few laboratories had the capacity to test by RT-PCR for SARS-CoV-2 as this method requires adequate infrastructure, equipment, and well-trained professionals. In addition to these factors, the high global demand for RT-PCR reagents led to shortages of the tools needed to perform this test, as a result, Peru was considered one of the countries with the lowest diagnostic testing capacity for COVID-19 in Latin America [4], with approximately 500 RT-PCR tests per day performed in a country with a population of 32.971.846 individuals. Posteriorly, the diagnostic capacity increased up to~10,000 tests per day but despite efforts to obtain the majority of molecular test results in the shortest period of time, the number of cases of COVID-19 was rapidly increasing and the RT-PCR testing capacity remained insufficient to test all the population who needed it. In this context, it became crucial to explore other diagnostic options, particularly as antibody based-RDTs were misused as a diagnostic tool for acute infection [5,6]. At the time of study enrollment, Peru was facing the beginning of the second wave [7]which allowed to evaluate the performance of Ag-RDTs in a real-life context, and test its potential as a rapid diagnostic tool, to facilitate epidemiological surveillance and contact tracing in low-resource settings.
In the UK, following the WHO and European Commission emphasis on testing to halt the spread of SARS-CoV-2, capacity for mass testing significantly increased towards~500,000 RT-PCR tests per day by November 2020, with a laboratory-based testing capacity reaching over 800,000 tests a day in January 2021 [8]. Through the UK governments' COVID-19 testing strategy, tests are available through NHS facilities, mobile testing units and satellite centres and includes 50 Regional Drive-Through COVID-19 Test Sites across the UK [8,9]. Trials for Ag-RDTs using the Innova SARS-CoV-2 Antigen Rapid Qualitative Test (Innova Medical group, UK) were introduced across UK hospital staff, Armed forces and School cohorts, with a city-wide mass testing implementation in Liverpool [10,11]. Whilst these studies provide an insight into the accuracy of such tests in the UK, challenges in testing capacity and Ag-RDT availability as well as differences in SARS-CoV-2 variant circulation and prevalence in other countries are likely to impact the agreement of findings and adoption of particular Ag-RDTs across global settings. Prospective diagnostic evaluation studies across multiple, independent sites are required to determine the accuracy of COVID-19 Ag-RDTs that are affordable and accessible to LMICs, in order to promote equitable access to reliable diagnosis and support containment of SARS-CoV-2 around the world.
This study evaluates the clinical performance of two brands of Ag-RDTs commercially available at the time of this study: GENEDIA W COVID-19 (Green Cross Medical Science, Korea) referred to as GENEDIA and the ActiveXpress+ COVID-19 Complete Testing Kit (Edinburgh Genetics Ltd, UK) referred to as the ActiveXpress+. The evaluated ActiveXpress + version is no longer commercialized. Both Ag-RDTs were evaluated against SARS-CoV-2 RT-qPCR testing in Peru and the UK in different settings: two primary health centers and one hospital in Lima (Peru) and a National COVID-19 Drive-Through Testing Centre publicly open to symptomatic individuals located in Liverpool (UK).

Study design, settings and participants
This was a prospective evaluation of consecutive participants enrolled in two different settings: In Peru, adults presenting with symptoms of COVID-19 were invited to participate in the study between January and March of 2021. Recruitment took place at two primary health centers and one hospital: Max Arias Health Center, Medalla Milagrosa Health Center and Huaycán Hospital, all located in Lima, Peru. Ethical approval was obtained from the Ethics Committee of Universitad Peruana Cayetano Heredia (SIDISI:202734) and was registered at the Peruvian National Institute of Health repository (PRISA) with the number EI00000001410.
Following all the safety precautions, trained clinical research nurses obtained the written informed consent form and started to collect two nasopharyngeal (NP) swab samples from each participant, one nostril using commercially available swabs and the other one using manufacturers swabs following IFU of Ag test. The first swab was used for the Ag-RDT that was performed immediately after collection of samples at the clinical site, where results were reading after 15-20 minutes following the instructions for use (IFU) by manufacturer. While the second NP swab was placed in universal transport media (UTM) Puritan1 UniTranz-RT Transport Systems (Puritan Medical Products, USA) and transported in coolers to the Laboratory of the Institute of Tropical Medicine Alexander Von Humboldt at the University Peruana Cayetano Heredia for the reference SARS-CoV-2 RT-qPCR testing. RNA extraction was performed using the Qiagen viral RNA extraction Kit (Qiagen, Germany). To amplify viral RNA, 8 μL was run on the CFX96 Touch Real-Time PCR Detection System (BIORAD, USA) using the Norgen's 2019-nCoV TaqMan RT-PCR Kit (Norgen Biotek Corp., Canada). Positive and negative controls were included in each run, as well as an internal control.
In the UK, adults presenting with symptoms of COVID-19 at a national community testing facility, the Liverpool John Lennon Airport Drive-Through COVID-19 test centre, were asked to participate in the study and provide a tacit consent. Participants were recruited between January and May of 2021 under the Facilitating Accelerated COVID-19 Diagnostics (FALCON) study. Ethical approval was obtained from the National Research Ethics Service and the Health Research Authority (IRAS ID:28422, clinical trial ID: NCT04408170).
Paired swabs were taken systematically by a trained healthcare professional, NP swab samples were collected to perform the Ag-RDTs and followed by combined throat and nose (TN) swab samples in UTM (Copan Diagnostics Inc, Italy) used for the reference RT-qPCR test, following the national standard of care. Swab samples were obtained by healthcare professionals and transported with coolers to the Liverpool School of Tropical Medicine (LSTM). Recommended swabs for each Ag test were processed by trained laboratory researchers according to the Ag-RDT IFU. Both Ag-RDTs were performed upon arrival, with a time delay of 1-3 hours, on the dry NP swabs, while TN swabs in UTM were aliquoted and stored at -80˚C until RNA extraction. RNA was extracted using the QIAamp1 96 Virus QIAcube1 HT kit (Qiagen, Germany) on the QIAcube1 (Qiagen, Germany) and screened using TaqPath COVID-19 (ThermoFisher, UK) on the QuantStudio 5 TM thermocycler (ThermoFisher, UK). Positive and negative controls were included in each run, as well as an internal control according to IFU.

Analytical sensitivity
A SARS-CoV-2 strain (202,012/01) from the B.1.1.7 lineage (Genbank accession number: MW980115), was used to investigate the limit of detection (LOD) of GENEDIA and the Acti-veXpress+. Frozen aliquots of the third passage of the virus were quantified via plaque assay. For the determination of the LOD, a fresh aliquot was serially diluted from 1.0x 10 6 plaque forming units (pfu)/ml to 1.0 x 10 2 pfu/ml. Each dilution was tested in triplicate. Two-fold dilutions were made below the ten-fold LOD dilution to confirm the lowest LOD. Culture media was used as negative control.
Viral RNA was extracted from each dilution using QIAmp Viral RNA mini kit (Qiagen, Germany) according to the manufacturer's instructions, and quantified using Genesig RT-PCR (Primer Design, UK). Genome copy number/ml (gcn/ml) were calculated as previously described [12].

Statistical analysis
The sensitivity and specificity, with 95% confidence intervals (CIs) of the GENEDIA and Acti-veXpress+ devices were calculated based on the results of the reference method by RT-qPCR assay. Statistical analyses were performed using R scripts and GraphPad Prism 9.1.0 (Graph-Pad Software, Inc, California). The 95% confidence interval (CI) for the sensitivity and specificity was calculated using Wilson's method [13]. Fisher's exact and chi-squared tests were used to determine non-random associations between categorical variables.

Clinical evaluation
The demographics of both the Peruvian and UK study cohorts are shown in Table 1. In Peru the median days from onset of symptoms was 6 days with no vaccination in Peru during recruitment. In the UK the median days from onset of symptoms was 2 days and a vaccination level of 21.15% was found across the combined cohort for GENEDIA and ActiveXpress+ evaluation. The clinical sensitivity of both Ag-RDTs were very heterogeneous between cohorts.
In Peru, of the 228 participants recruited, 108 samples were tested with Genedia and 120 with ActiveXpress+ Ag-RDT. Specifically, for GENEDIA W COVID-19 Ag Device evaluation, fifty-four of 108 (50%) samples were positive by RT-PCR (See Table 2). Thirty-nine of the RT-PCR positive samples (72.22%) were Ag-RDT positive, while the remaining 15  In the UK, of the 615 participants recruited, 399 samples were tested with Genedia and 212 with ActiveXpress+ Ag-RDT, four participants from Genedia group and one from ActiveX-press+ were PCR undetermined and excluded from all analysis due to not have paired data. Ninety-five (23.8%) of 399 specimens collected during the enrolment period of the GENEDIA evaluation were positive for COVID-19 by RT-qPCR (see Table 2). Fifty-one of the RT-PCRpositive samples (53.7%) were Ag-RDT positive, while the remaining were Ag-RDT negative. Of the 304 RT-qPCR negative specimens, only 2 were Ag-RDT positive. This UK evaluation of the GENEDIA Ag-RDT showed a sensitivity and specificity of 53.7% [95% CI 43.1-64.0%] and During the enrolment period in the UK for the ActiveXpress+, 11 of 211 (5.2%) specimens were positive for COVID-19 by RT-PCR (see Table 3 Subgroup analyses of the Peruvian and UK evaluation cohorts (Table 4) were performed to determine any associated differences in viral antigen detection/sensitivity compared to vaccination status and days from symptom onset to test. No discernible differences in viral antigen detection/sensitivity were detected for days from symptom onset to test and participants who were vaccinated vs non-vaccinated or those who received 1 dose vs 2 doses of the COVID-9 vaccine in the UK for either evaluation cohorts of the GENEDIA or ActiveXpress+ (all p values >0.05), the subgroup evaluation for Peru´s cohort was not performed due to not availability of vaccines in the country at time of enrolment.
Also the values of Likelihood ratio + and-were calculated, for GENEDIA the overall LR + value was 72, and overall LR-has a value of 0.39; while for ActiveXpress+ the LR + value was 220, and overall LR-has a value of 0.33. In both brands the values of Likelihood ratio were highly relevant to consider the COVID-19 diagnostic if a participant had a positive Ag-RDT result.

Analytical evaluation
The LOD of GENEDIA W and ActiveXpress+ was 5.0 x 10 2 pfu/ml which equals to approximately 1.0 x 10 4 gcn/ml fulfilling the recommendations in the WHO Target Product Profile for SARS-CoV-2 Ag-RDT [14].

Discussion
The study aimed to evaluate the diagnostic performance of GENEDIA and ActiveXpress+ Ag-RDTs. Determining the diagnostic performance of Ag-RDTs is important because this gives Table 3 an indication of their clinical utility in a pandemic setting, as such tools could improve the time of response and clinical decision making. An adequate sensitivity and specificity values may lead to an increase in testing capacity in low-income settings, particularly where no RT-qPCR can be deployed, and simplifying contact tracing. A Cochrane systematic review of 48 study cohorts which assessed Ag-RDT performance reported a paucity of high-quality data from prospective studies. Of the evidence identified, sensitivities of between 34.1% (95% CI 29.7% to 38.8%; Coris Bioconcept) to 88.1% (95% CI 84.2% to 91.1%; SD Biosensor STANDARD Q) were reported in symptomatic participants (15). Data of previous field evaluations of Ag-RDT shows variation in sensitivity, in Uganda the STANDARD Q COVID-19 Ag test showed a sensitivity of 70% and specificity of 92% (16), and, in Spain, an evaluation in primary health centers with Panbio™ COVID-19 Ag Rapid Test Device showed sensitivity and specificity values of 79.6% (95%CI 67.0-88.8%) and 100% (95% CI 98.7-100%) respectively (17). Also, an Ag-RDT evaluation in Egypt with SD Biosensor STANDARD Q [18] was made entirely in a laboratory setting and showed sensitivity of 78.2%, the sensitivity values varies from study to study which could be explained by variation in sample collection and epidemiological characteristics of different populations [14][15][16][17][18][19][20]. Of many Ag-RDT performance studies previously carried out, sensitivity values are found to be consistently higher for Ct <25 [15][16][17][18].

Results of ActiveXpress+
In this study, the overall clinical sensitivity using GENEDIA (60.4% [95% CI 52.4-67.9]) and ActiveXpress+ (66.2% [95% CI 54.0-76.5]) was lower than reported from manufacturer (GENEDIA: 87.25%, ActiveXpress+: 90%), values which are below the recommended sensitivity for Ag-RDTs according to WHO [8]. Also, GENEDIA brand showed better sensitivity for SARS-CoV-2 detection when RT-qPCR had Ct<25, the overall of UK and Perú cohort for these Ct values was 67.5% and superior if compared with the overall sensitivity for this brand, which is concordant with WHO recommendations for Ag test use [8] and with previous studies [17][18][19][20]. Also, for the ActiveXpress+ evaluation, there is a difference between Peru´s (61.0%) and UK sensitivity (90.9%), that could be explained by the number of participants with positive results enrolled for each cohort (54 and 11 in Perú and UK, respectively) due to overestimation of sensitivity in context with low prevalence as UK, as well as the differences in the type of samples used for RT-PCR (NP swabs vs. NP swabs and NT swabs) as the amount of RNA collection is dependent of the site of sample collection and these RNA values are strongly related with sensitivity of PCR assays (19)whereas sample processing of Ag-RDT test was performed in different scenarios. In Peru, Ag-RDTs were undertaken by clinical research nurses, whereas in the UK the tests were run by laboratory scientists. Previous research has suggested that the sensitivity of Ag-RDTs is higher when run by laboratory scientists than when run by healthcare professionals. For example, the Innova Ag-RDT was shown to have a sensitivity of 70% when run by healthcare professionals, versus 79% when run by laboratory scientists [10].
The higher sensitivity with low Ct values is a constant observation between previous evaluations performance of Ag-RDTs in patients with high viral loads and it is relevant as several studies associate high viral loads with worse prognosis and high rates of severity disease [19]. Moreover, these patients are more infectious, and their early detection could facilitate epidemiological surveillance and contact tracing [20].
Interestingly, as the median days from symptom onset increased, the sensitivity of both Ag-RDTs decreased in Peru's cohort. Similar to previous evaluations of Ag-RDTs where sensitivity decreased among symptomatic patients when compared with testing within 5 days and >5 days of symptom onset [21]. However, in the UK cohort of this study, this trend was not observed for either Ag-RDT, with no significant difference in sensitivity at <0-3 days vs 4-7 days from symptom onset found during the ActiveXpress+ evaluation, also there was no specific trend of higher sensitivity values related with days of symptoms in GENEDIA brand for UK cohort. Studies have found the SARS-CoV-2 viral load to increase over the course 5 to 7 days which is generally observed to be the optimal point regarding RT-PCR sensitivity [22]. In this cohort it is likely that the <0-3 days subgroup, did not yet have the viral loads for optimum detection compared to the 4-7 days subgroup.
The circulation of several variants and the prevalence at time of recruitment could affect the observed performance of the Ag-RDT in the two different settings. In Peru, the Alpha (B.1.1.7) variant was first reported in December 2020 [22], the enrolment of participants for GENEDIA started on January 2021 and for Active Xpress+ on February of the same year, during this period, Alpha prevalence was low (1.5%), with the Lambda (C.37) lineage the predominant variant in circulation [23]. Whilst in the UK, Alpha (B.1.1.7) was dominant at the start of the recruitment period in January 2021, with the Delta variant (B.1.617.2) variant rising to a frequency of 46% across the UK by the end of May 2021 [24,25].
For RT-PCR sample type, the standard of care in the UK recommends combined nose throat swabs for its national COVID-19 testing program. In Peru, the National Institute of Health recommends combined Nasopharyngeal and Oropharyngeal swab for COVID -19 testing. Literature suggests that Ct values and load of SARS-CoV-2 detection in swabs are affected by the collection site, due to high quality and relevant abundance of RNA depends of specific site of collection and this is crucial for the sensitivity of the assays [19]. Regarding type of samples collected by both cohorts in this study, while in Peru only NP swabs were performed for Ag-RDT and RT-PCR, in the UK NP swabs were used for Ag-RDT and nasal and throat swabs for RT-PCR. Reports in late 2020 indicated that nasopharyngeal swabs (NP swabs) were the most specific and accurate swab site for COVID-19 diagnosis, followed by throat swabs [23]. Later research included combined nasal and throat swabs as an additional comparator, showed a higher positivity rate of 100% compared to nasopharyngeal swabs (91.5%) on RT-PCR for SARS-CoV-2 RNA detection [24]. Additionally, the Peru sampling used two NP swabs for Ag-RDT and RT-PCR with unknown comparability between swab types; it is undetermined whether this had an effect on the viral load. With all these considerations, including the difference of sample and RT-qPCR assay type along with the circulation of different variants and differences in recruitment settings between these two cohorts, the variation in sensitivity values of each clinical site would be explained.
Regarding vaccination status, at time of study development, in Perú none of the participants received any doses and data about previously confirmed SARS-CoV-2 infection was not available. The UK cohort included participants who had received vaccination for COVID-19, vaccinated individuals were defined as a person who received any dose of any vaccine brand. When analysed for differences in sensitivity, no significance was detected between the vaccinated and unvaccinated groups. However, an evaluation of effectiveness of COVID-19 vaccines among Health Workers in Israel including the comparison between Ag-RDTs and RT-PCR results, all the participants with positive RT-PCR result undergoes to Ag-RDTs evaluation, were the Ag-RDT positivity for unvaccinated, partially vaccinated and fully vaccinated was 80%, 33% and 21% respectively [25]. Further analysis is warranted to observe whether vaccination affected the viral load detected by Ag-RDTs.
Given the WHO considerations for minimum performance requirements include a sensitivity >80% and specificity values >97% [14], both tests have less-than-optimal performance. Further investigation into the application of these Ag-RDTs across different settings and populations at higher prevalence may provide setting-specific or prevalence-specific performance of SARS-CoV-2 antigen detection and as such further implementation studies are recommended. Edinburgh Genetics have taken Active express off market and are now distributing a new product.
This study had some limitations. First, no follow-up of participants was performed to confirm COVID-19 disease severity, the study was intended to evaluate the usefulness of these diagnostic tool to yield results in the shortest time and with the greatest efficiency, for this reason the specimens were taken at a single time point without perform a second Ag test that may increase assay sensitivity which represents another limitation.
Another limitation is the heterogeneity of participants between both cohorts regarding vaccination status and specimen testing relative to the last vaccine dose. Also, the difference between the rates of positivity on each country at time of study development, while in Peru the study was performed during the peak of the second wave, in UK only a small number of positive samples were included in the UK evaluation owing to the relatively low prevalence in Liverpool during the recruitment period, these differences could lead a bias in the interpretation of results and overestimates sensitivity values in a context where the prevalence was low as UK. In UK cohort, the PCR specimens that had undetermined results were not tested in both platforms due to all the non-paired data were excluded from all the analyses.
Finally, the comparison of select specimens with viral culture to determine the proportion of replication-competent SARS-CoV-2 isolates that tested negative by Ag-RDTs would have been an important outcome to report, and could be considered as another limitation of this study, we suggest considering this for future performance evaluations related to Ag-RDTs.
To date many evaluations have been restricted to single-centre studies or within one geographical region. This study presents a multicenter evaluation across two different settings with different testing capacity, access to vaccines and prevalence of SARS-CoV-2; in the UK, the introduction of mass vaccination during lowering prevalence, and in Peru, a less controlled increase in positive cases, despite having the world's highest number of deaths per capita at the time of study [26]. With all these considerations, this study is intended to contribute to gather independent data on the performance of COVID-19 Ag-RDTs that are much needed to provide a rapid and accurate detection of SARS-CoV-2 infection to support already stretched healthcare systems especially in LMICs.
In conclusion, our data indicate that the GENEDIA W COVID-19 Ag Device and EDIN-BURGH Genetics ActiveXpress+ COVID-19 have poorer performance to that published by manufacturers for the detection of SARS-CoV-2 from clinical samples. Resources: Sandra Palomino-Padilla, Luz Villa-Castillo, Eloïse Cook, Camille Escadafal.